Empirical Bayes for Learning to Learn

We present a new model for studying mul-titask learning, linking theoretical results to practical simulations. In our model all tasks are combined in a single feedforward neu-ral network. Learning is implemented in a Bayesian fashion. In this Bayesian framework the hidden-to-output weights, being speciic to each task, play the role of model parameters. The input-to-hidden weights, which are shared between all tasks, are treated as hyperparameters. Other hyper-parameters describe error variance and correlations and priors for the model parameters. An important feature of our model is that the probability of these hyperparam-eters given the data can be computed ex-plicitely and only depends on a set of suu-cient statistics. None of these statistics scales with the number of tasks or patterns, which makes empirical Bayes for multitask learning a relatively straightforward optimization problem. Simulations on real-world data sets on single-copy newspaper and magazine sales illustrate properties of multitask learning. Most notably we derive experimental curves for \learning to learn" that can be linked to theoretical results obtained elsewhere.